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ABSTRACT * - 

This paper, designed to assess the degree of 
implementation of the important features of a curriculum innovation, 
was presented at the 1978 Bat-Sheva Seminar on Curriculum 
Implementation and its Relationship to Curriculum Development in ^ 
Science, Rehovot-Jerusalem , Israel. It investigates the use of 
sub-interViews within the main interview to assess the lievel of use 
of each of the components of interest innovation bundle. The 
methodology described is used to measure the level of implementation 
of the junibr high school science program. The National World" (TNfi), 
Tjhis is the revision and modular .adaptation of the Intermediate 
Science Curriculum Study (ISCS) • A total of five teachers using the 
TRW program were interviewed personally or by telephone. Results show 
that all teachers are regular users of the program and there are 
considerable variations in their degree and level use of the program. 
Scales ""used to measure the use of innovation and interview format are 
also ^)resent€d, (HMJ 
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ASSESSING THE DEGREE OP IIfl>LENEMTATlON OP THE J' 
IMPORTANT -lEA'pOBES OP A CUpioULAH INNOVATION 



; Any review of the lit'erattlre an education' reveals'*toany studies ^ ^. 
.that repoart no significant findings^ or indicate conflicting resiilts. The 
■llacH of definitive findings regarding the superiority of pajrticular .pro- 
grams, material^, s'trategies,' and techniqties has firustrated researchers 
and practitioners alike for years. • A few researchers such as Stephens 
(1967) seem' to have despaired -of ever obtai^ng definitive refults. Some 
■ more optiiQis-6ic researchers have sou^t achievement- treatment interactions. 
Although this research area has yielded "some results, progress has been 
slow (Tobias, 1976), Other researchers have' questioned the basic research 
methodologies used d.n educational ^'research. 

Charfers and- Jones (1975) (i^^^ew the attention of the research com- 
taunity to the fact that sometimes the differences between "experimental'' 
^ajid^ "control" programs gtre more fictional than factiial. They stated t"hat 
unless adequate measiirement techniques are used this lack of difference 
will "be undetected. - ' ^ \' p 

The relevajfice of this to curripulum research was shown-^ty Hall 
and Loucks (1977) • Their study compao^ed student achievement; in reading * 
and mathematics- between schools using: Individually Guided Education (iGE) • 
and hon-IGB schools. Initial results indicated no significant differ- 
ences. However, .when\ the researches ixfterviewed teachers, to determine 
the extent to which each teacher was individualizing reading and math- • 
ematics in./:their classroom, they found that use and nonuse of individual- 
ization wa» widespread in both IGE and hon-IGE schools. Re-axialysis of 
the data to compare kctual users luid^nonusers of individualization 
revealed significant differences. This study clearly indicated the need 
for careful measurement to ensure that curriculum innovations have been 
implemented in the "experimental" group ajnd not in the "control" group. 

A number of different techniques have been used to assess the 
fidelity of implementation, the degree to which the actual use o^f Vn in- . 
novation corresponds to its intended or planned use. These include: 



direct classroom otservation (Ashley and Butts., 1970; Evans and Scheffler, 
1*974; Gross, Giacqxxinta, and Bernstein,. 1971; Hess and BucMioldt, 1974; ' 
and Naiimann-Etienne, 1974) > a comtination of questionnaires' and. direct 
observation (Crowther, 1972; Leinhardt, 1975; Leinhardt, 1974), question- 
^naires and an analysis of ^ey documen-ts (Downey et al,, 1975) > ^teacher 
questionnaires (Cole, 197i)> axid focused interviews v^ith teachers (Hall 
and Loucks, 1977). 

Direct classroom observation over an. extended period v;ould seem 
to "be the most desirable and valid technique of assessing the degree of 
implenientation of a curricular innovation; however, - limitations of time 
and money often preclude the use of this ^echnique. 

The Level of Use framework (Hall ,6t al. , 1975; Hall & Rutherford, . 
1976) appears to be an attractive alterative. This framework is j8ased 
on the Concerns-Based Adoption Model .^CBAM) developed by Hall, Wallace, 
and Dossett (1975). They hypothesii^ed eight different Levels of Use: . 
Non-use, orientation, preparation, /mechanical use, routine /^use_7, 
refinement, integration, and rer^C/al. A branc^^^P^^ocu3ed interview 
with the user of the irii^t)vatioxi/( teachei?) is uped to assess the Level of 
Use of the innovation j'n'the olassrooii^^(Loucks , Nevrlove, '& Hall, 1975V 
Evaltiations of the tebhnique/indicated. interrater reliabilities of from 
0.87 to 0.96 for repeated :^tings of .tape-recorded interviews and a cor- 
relation between intervie^ ratings ' and ratings by ethnographers, v;ho 
spent a full day v;ith a ^^^ple of teachers, of 0.98 (Hall & Loucks, 1^77). 
In addition, longitudir^l studies indicate that there^ is a gradual molre- 
ment of users from lo\^pr levels of use to hi^er levels of use as a func- 
tion of the length ow time an innovation has been used (Loucks, 1978). 

The compreher^siveness of this framework is. an asset to the re- 
searcher who is stipying the implementation process , in depth. But, the 
complexity of the/scale and the measurement technique may prove to be a 
liability/to tlie /researcher vho is only interested in assessing the 
relative/ ^'purit^" of experimental and control groiips. This paper presents 
an abbreviated/ form of the scale for use in such situations. 

A second drawback of the original framework in research on cixr- 
ricula is th^t it is designed to measure the level of use of a single 
innovation ^vhile a t^ical curricular "innovation" actually consists of 
a "biandle"/of independent processes and ideas. The Level of Use Training 
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W^£ual recognizes this problem but the ou^gestion of using oeparate, 
domplete interviews for each or the separate innovations is not entirely 
satisfactory (LoucIcs, Uevlove, & Hall, 1975, PP. 30-52). This paper 
proposed the use of "sub-interviews*' within the main interview to assess 
the level of use of each of the components of ^iijterest in the curriculum 
innovation bundle, 

A more s'erious problem in the uke of the original level of use 
frainework is that there 1b no formalized strategy for assessing the^ 
extent to which the innovation is used in the classroom. The importance 
of this cisjiect'of measurement is emphasized by the study of Gross, 
Gia!cquinta, and Bematein (19.71) in which direct classroom observation 
indicated that teachers behaved in accordance with the expected role only 
about twenty perclfent of the time. G?hese teachers may hav^ been implemen- 
ting the iimovation at a mechanical level or even a routine level but the 
extent of their implementation was low. Loucks, Newlove, and Hall (l975, 
pp. 51) "^se one or more questions at the beginning of the interview to. 
assess the extent of use of the innovation but only to separate users 
from nonusers. This paper expands on this technique to assess the extent 
of use of the total curricul-um package and the various separate innova/- 
tions of interest. 

Levels of Curriculum Implementation 

* There appear to be two independent factors that mnist be considered 

in ■accounting for the variation observed in teachers* actual use of a 
curricioliom innovation in a classroom situation. The first is the fidelity 
of uoe of the innoyatidn or the degree to which the original intentions' 
of the instructional designer are followed in implementing it. The sec^ 
ond is the degree of use or the proportion of appropriate instructional 
time that is devoted to the use of the innovation. 

The fidelity of use factor is very difficult to define and measure 
ac curat elji^ In their review Pull an and Pbmfret (1977) indicate that the 
measurement of levels of use of an innovation proposed by Hall and Loucks 
(1977) takeii^the measurement of the fide;Liti/^f implementation to its 
logical and methodological conclusion. This method, modified to reduce 
its complexity and to permit the sepcirate assessment of the various sep- 
arate assessment of the various separate innovations inliercnt in a cui>- . 



riculum innovatior;, is the one proposed in this paper. 

The degree of use of an innovation is also difficult to measure 
but it must be considered. Teachers vary in the .e::rfcent of their use of 
axx entire curricul'uin package. Some teachers v^ill use one curricTzlar 
package throughout the year with all of their students. Others will 
limit the use to certain students or certain segments of tlie course. 
Teachers also vary in the extent to which they implement the various 
features features or innovations inherent in a curriculum innovation. 
This is a particularly important consideration in such recent curricTila:^ 
innovations as the Intermediate Science CurricTzliim Study (iSGS) material 
and the Inij-vi dualized Scie^iice Instructional System (iSIS) materials 
which are specifically designed to provT&e the teacher with the flex- 
ibility of implementing or not implementing such important features as: 
self-pacing, self-sequencing, self-diagnosis and prescription of remedia 
tion, branphing, and self-evaluation. Thus it is necessary to measure 
both the aegree of use of the entire curricTilixm package and the degree 
of use of the various separate features or innovations inherent in the 
package to account for the variability encoimtered in actual use. 

The metl^odology d^'scribed below uses separate scales to me^^^lire 
the two factors: the degree of use and the level of use. These tvra 
scales form a degree of implefiaentation (DoI) matrix. One Dol matrix . 
is produced for the overall program and one for each of the innovations 
inherent in the program. ^ 

/ ' ■ , ' ' . ^ 

I)#$^tion Qf the Modified Level of Use (fr3>g-Mod) Scale 

The fidelity of use of an innovation is measured using a modified 
'^form of the Level of Use (LoTJ) Scale described by Hall et al. (l975) an<i 
. Loucks (1977). The modified Level of Use Scale (LoU-Mod) distinguishes 
between four discrete levels of use of a curriculum innovation: mechan- 
ical use, routine use, refinement, and modification. The scale points 
of the LoU (Mod) are defined in Figure 1. The' first three levels are 
essentially the same as those of the original LoU Scale. The fourth 
levels modified use, is used to characterize situations in v;hich the 
user has made substantial changes in the overall program or innovative 
feature so that the instructional intent is altered. 
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/ Figure 1: Definition of LoU-Mod Scale Points' ' ^ 



LoU-Mod Scale Points . ' Definitions 



Level 


A 


- Mechanical 


The state ^^j^which the user is prilaxily 
. concerned a^bout the day-to-day mechaiiics 
of use» There is little time for reflec- 
tion. Ch^ges are made to meet user needs 
rather than student needs. 


Level 


B 


- Ro-^ine • 
^^^^ 


The state in which the use of the program 
or feature lias stabilized. Few changes 
are made, o Little thougljt is given to im- 
proving its use or impact. 


Level 


C 


- Refinement 


The state in wlaich the use is varied to 
, Increase its impact onT students. The var^ 
iations are teased on the use3?s knowledge 
of the short and long tem effects of the 
program or feature on students and its 
demands on them. 


Level D 


- Modification 


The sljate in which ^the user has substah- 
tially modified the program or feature so * 
that it no longer me6ts the original 
intent. 

I 



Several scale points of the' "original LoU Scale have been deleted 
or shifted to the degree of use ..dimension. The scale points "non-use" 
and "integration*' have' been shifted to the degree of use scale. The 
scale points "orientation" and "preparation" have been deleted since they 
represent various levels of preparation for use rather than act^ual "UBe 
and so are of little interest to a researcher attempting to assess the 
"purity" of th^' experimental and control groups. Another scaj^e points 
that has been deleted is "renewal". This scale point is ^ used to charac- 
terize an individuai^ who is ciirrently using the innovation but who is 
seeking alternatives rfco replace the innovation. Since such a person is 
still actively using rche innovation in the classroom, the value of iden- 
tify^ing this categorylof users seemed marginal and this scale point v/as 
deleted from the LoU-J4od. 

There are a variety of behaviors that are characteristic of the 
various levels of use of an innovation. The original LoTJ Scale describes 
seven different categories of behaviors that vary from level to level. 
These categories are: knowledge, acquiring information, sharing, assess-* 
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Vigure 2: The LoU-Mod Chart 



DEFINITION OF livEL OP 
. IJ'SE SCALE ms 


BEHAVIORi 


i CATEGORIES 


INPOBltolON 'SEEKING 


EVALUATIVE BEHAVIOR 


STATUS IffiKlRTING 


LEELA--MECMICAL 
The user is primarily 
concerned atout the . 
day-to-day mechanics 
Oi use. iiitxie tune 
for reflection, Changes 
made to meet user needs 
raffia than student , , 
n^eds. 


Seeks information about 
management-related 
topics as scheduling, 
'timing, snd logistics 
anfi lueas w reuuce .une 
time and effort required 
of the user. 


Evaluates ^ise with . 
respect to problems of 
logistics, management, 
time, resources, equip- 
ment, sch^dilles and 
general reaction of 
students, 

-• * 


Reports that logistics, • 
tiije, scheMing,resoui'ce 
organization, etc, are the 
focus of most personal ef- 
forts to use the program 
or feature, 

\ 


LEl/EL B - EOUTIIE ' 
The use of the prograia 
or feature has stabil- 
iaed, Pew changes are 
made, little thought is 
given to improving its 
usQ of imijact. 


Makes no particular 
effort to seek new in- 
foimation or ideas. 


Limits evaluatiqa to - 
that which is admini- 
■ strativeiy required. 
Mttle attention paid 
to, findings for the 
purpose of changing" 
use. 


Reports that personal use 
is proceeding satisfactor- 
ily with few, if any, . ^ 
problems. . 

, ) 


LEVEL. C - BEPIMTO . 
The use is varied to in- 
crease the impact on 
students. The varia- 
tions a]je based on a 
knowledge of the short 
and .long-term effects 
and demands on students,. 


^eeks infonnation, 
idea!^, and materials to 
serve as the basis for 
cJianging use lo aiieoi; 
student outcomes. 


Evaluates om use of 
the ctratefy -or 'program 
for the purpose of 
changing current prao- 
tices to improve stu- 
dent outcctaes. 


' 1 

Reports vary;Lng program or 
feature, in order to modify 
student outcomes, 

I 


LEVEL D - KOBIPICATIOU 
The user has substanti- 
ally modified the pro- ^ 
gram or^ feature so that' 
it is no longer the 
same as the original, ; 


, Seeks Infomation, " 
, ideas,' and materials to 
implement major changes, 


Analyzes strendihs and 
wealaesses of 'current 
practices for the pur- 
pose of implementing ' 
major changes. 


Reports substantial change 
and modification to the 
program or feature, 

> 



NOTE: A modification, of the original Lol Chart devised by Hall et al, {l9]5) 
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ing, plajining, statuB teporting, and perfonning.^ Tho simplifiod LoU-Mod . 
Scale retains only three of these categories: infonaation eeeking, eval- 
uative behavior, and status reporting* Th^ various Crategories and behav*^ 
iors characteristic of each level aai^B^^Dummarized in Figure 2. The behav- 
loro outlined in this figure form the basis of the assessment ~of an 
individual's level of use of an innovation. The focused interview, do-:^ 
scribed below, is d(?signed to elicit responses that describe the respon-^\ 
dent's typical behaviors :ui each of these behavioral categories axid thus) 
allow the interviewer to determine the actual level of use of the innova-;^ 
tion by the respondent. ' * jt 



The LoU-Mod ScaJ.e is used Ho describe the level of overall use of 
a curricultun package and the levels of use of the various separate inno- 
vations or features that are inherent in the curriculum package. / 

' ■ % 



Definition of the De^^ee of Use (l)oU) Scale n ?^^''^ h 

The variation in the extent to which a curricxilar j^pgrara or J 
feature is used in the actual classiroom situation is describ^. wjbjp^.^pr^ ""^'"^ 
to the Degree of Use (DoU) scale* The extent to which a teacher 'actiiatly 
uses an innovation is essentially independent of that teacher's level of ' 
use of that innovation. For emmple, when^ a teacher implements an innova- 
tion such as self-pacing of instruction the teacher may demonstrate a 
routine le^rel of use of that innovation, but for one reason or another may 
choose not to implement that innovation in the classroom on a regular basis. 

The scale points of the DoU scale are defined in Figure 3 and range 
from non-use of the innovation to integrative use* The selection aiid defi- 
nition of these scale points are arbitrary and they; v/ere chosen so as to 
keep the scale as" simple as possible yet provide the necessary information 
for a researcher interested in determining* the "purity" of experimental 
and control groups. The "partial use" scale point ca,n easily be separated 
into three scale points to indicate; regular use by a portion of the class, 

irregular use by a portion of the class, and irregular use by the entire 

\ 

class* ^ 

T\7o of the scale points, "outside use" dnd "integrative use," . require 
fiirther explanation. These apply only to the individual innovative curric- 
ulai^features and not to the cujrriouluiQ package as a whole* They are used 
to describe situations^ in which students are exposed to the innovative 
curricular fe««Jures of interest in classes other than the one that is the 
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DoU Scale Point n > 



Iiovol 0 - Non-Uno 



Tiovel 1 - Outr.ido Utjo 

(for pro^jxain featufoii 
only) 



Level 2 - Partiul Ur3o 



Level 3 - Hcifculru?, U^e 



J)o.fln.LLion of DoU fJcalo rolntu 
' Deflnil. Lgno 



/Level 4 - Inte^prative Uoo 

)(for i:)ro(;rain featiLirer-^ 
I only) 



1^0 prof;cfiin or feature ia not unod at all. 
Thin ncale point incliidon pnBt Uf^ertj vlio 
have no definite plaJio to ubo it on a 
perloclic banio* J 

The feature in not uried in^ the c^usoroom 
in i/liich the pro^^rcim ia implemented hut 
the iitudentG are exponed to the feature 
in other clapines thoy attend, 

} _ 

Tlie pro-am or feature in used to a lim- 
. ited extent, A pro^^yram io utied by less 
thJin W/o of the utudentg or for leoo than 
IW/o of the available int^txuctional time. 
A feature io uaed with Icon thcUi iW/o of 
the utudenta or; in leoa thaii Ca/o of the 
suitable Dituatlons. 

llie pro^^\carQ or feature io uL;ed on a re^j;- 
ular, continuin,[^ baoio v;ith at leant W/o 
of the- Qtudenta and for at leaot OCk/* of 
th^available instructional, time (pro- 
^ama) or suitable situations (features). 

The featLU'o in iiaplemented in the^clarva- 
room at level 2 or 5 and the ctudenta are 
further exposed to the feature in other 
clashes they attend. 



main Tocua of intereat. Tho ■agefiilnons of thet'.e scale points can 
boot illuntrated usi.nf: an extunple. Suppose a reLsearclier vraa Btudying 
the effect of a acionce imit on fctudento' critical tbinlcin^' nkilla. His 
ronul U Wad bo meanin^^leaa if '.tie failed to 'ensure that attxdents in the 
expDrimerital ;md control fi-roups wfere not receiving: inatmiction m on t- . 
ical thinking nkiils in- their otl-ier- .claasoa . 

■ DotexTOxniiU? the do^a'oo of-\i3te'^c^f«r_an innovative feature can bo 
quite troublesome nince certni^i imio\^t%i.p. are not mcaxit for continuous 
claosroom uoo but rather when tho. sltivatlon arloca. '/FOx- ^example, tho 
celf diaV;no3i3 ajid prescription feature of tlie ISG^ pro-rcuu ia 'only ^ 
implemented when a atudent roacyies the appropriatV-^oction of the -in- 
structional materialr.. Clnsr.rooia obr.orvat.i on in a very inefficient 
mdihod of aipaesnin- the decree of Ur.e of sucli Jniiovativo features. An , 
interview with the teaclier or a at^ont Ih much |nore efficient but the 
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Interviewer iiiugt have the expected level of use of the innovation clearly 
in mind' to be able to i^ate the actual degree of use correctly. 

MeaGurinf^ Levels of Curricular Implementation ^ 

. Mea^^Lring the level of cii^icular implementation involves three 
Stages. Tliege are: Belooting 3^d operationally defining the separate 
innova^tiony qf interest v/ithin the ciu'riculuia package., interviewing tW 
t<^Nac hers, and rating the\ degxee and level o.f implementation. The proce- 
dures for each stage are\outlined below. 

Pefinin^n^ the Sepal^ate Innovations of Interest ' 

Curriculiini innovations, particularly those developed with public 
rather than private suppoi-t, tend to be relatively complex, 'incorporating 
oever^ ijinovative features, which distinguish them from more traditional 
pro^fcams. Tliese innovations range from content and content emphasis 
tWrou^ classroom organization and climate -io n-ew roles for the students 
afrid teacher. ' ^ ' ' 

Descriptive literature and teachers gjldes are often the best ref- 
erences for the researcher' in determining' which specific innovations are 
inherent in the curriculum package and in defining^ them. Hov/ever, the 
researcher is v;ell advised to 'talk v;ith users of the package and spend 
some time in a classroom where the package is being implemented because 
there raa-^ be x^ignificant discrepancies between the developers original 
intentions and the final product. / ' " \ 

In most cases an analysis will indicate that the curriculum 
package incojcpoi^ates a large number of separate innovative features. In 
these situations the researcher must be selective and consider only those 
feati'ires that may have a significant effect on the dependent variable in 
question*. The focused interview v;ill become too long^- and cumbersome if 
i^ includes questions relating; to more than five sepax\ate- innovative 
featirros. ' . ^ / ^ 

Tlie researcher must be careful to operationally define the innova^ 
tion to be assessed in terms of specific classroom behaviors. Unless- the 
interviewer has a clear idea of .v;hat the researcher means by the innova- 
tion and co^ communicate this to the teach^ being interviewed, the sub"- 
sequent rating of the degxee and level (pf )a3e will be meaningless. How- 
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ever,^ the definition must not "ba.^too tight beicause there is- al Ways a ceiv- 
tain'aiuoiint of modification 6f aji innovation , when it is implemented 
(Fellan & Pomfret, 1977, -p. ^357). ^ , . ^ " 

Conducting the Interview 

^ The degree and level of use of a total curriculxnn package and its 

innovative features are assessed using information supplied by the teacher 

i 

during a focused interview. This interview uses a. branched foi^t and, 
with a skilled interviewer, resembles a casual conversation with the 
teacher about how the program , is working in the classroom. 

The interview consists of two major parts. The first relates to 
the general level and degree of use of the entire cujrriculum package. 
The second portion deals with the specific features or innovations of 
interest axid is repeated for each of them. The suggested interview pro- 
tocol is given in Figure 4. The wording of the questions and transition 
statements need not be followed exactly but the interviewer must be suf- 
ficiently knowledgeable about the rating procedures to probe more deeply 
when the appropriate information has not been forthcoming. An annotated 
transcript of a sample interview is provided in Appendix 1. 

The length of an 'interview varies from 10 to 35 minutes depending 
on th^ number of specific innovations ecssessed, the talkativeness of the 
respondent, the skill of the interviewer, and the degree and level of 
use of the innovations. The interviewer should, if at all possible, 
tape record the interview so that more than one individual can rate the 
degree and level of use. 

Tlie training manual for the originali LoU scale (Loucks, Newlove, 
& Hall, 1975) is recommended reading for researchers who plan to make 
use of this technique. It contains many valuable ideas and suggestions. 
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Figure 4= Lol Interview Schediile 



Are ' 
'you ctirrently^ 
fusing the program^ 
with your 
^classes^ 

Do 
you use 
^it on, a regular 
clay-to-day "basis 
/ith all your 
^students 



Yes 



How much do 
you use it? 
(Probe for: 
proportipn 
of studerits 
using it & 
time spent) 



Generally speaking, how do you think 
the program as a whole is working in 
your situation? Any problems? If so, 
v;hat kinds? ^ 



J 




Are yoa looking for any new ideas or 
information about the program? If sc^ 
v;hat kind? Por what purpose^T 



Have you made any changes in how you 
use- the program? \^iat? V/hy? How 
recently? Are you considering making 
axiy (other) changes? 



What do you see as the effects of 
the px^ogrcun? How did you determine 
this? Have you gotten aiiy feedback 
from students about it? VJhat hsive 
you done with the information you 
obtained? 



TRANSITION : You've given me a clear 
idea of how you're using the program 
in general terms. Now let's foQUs on 
some spe cific features of it. . 



Do you plaji 
to use this 
program in 
the' future? 
If so;>wllen? 



TRANSITION ; 
The prog'rajn 
uses some 
interesting 
approaches 
to teaching 
you' may be 
using now. 
Let's see 
if you are. 



How long did you use it? How 
did you implement it in your 
classroom? \^iat problems did 
you find? V/hat effects did 
it seem to have on students? 



\Jhy flld you stop using it? 



V/lien you assess the program 
at this point in time, v;hat 
do you see as its strengths 
and v;ealaiesses? 



TRANSITION : ' An you laiow,the 



program has some interesting 
features. Let's see if you 
are still using some of them, 



4 



1^ 



One (ojiother) feature of 
the program is: (name)- 
In practice this means 
that : ( brief description ) 



Figure 4 (continued) 
Features Brief Description 



Are 

^ou curi^ntly^^ 
^sing this approach" 
V7ith your 
classes 

^Yes 



Do students 
you teach 
encounter 
it in other 
classes? If 
so, how much 
and where? 



Do 
you use 
^it on a regular^ 
continuing basis 
with each 
student 

9 

^Yes 



No 



How much do 
you use it? 
( Probe for: 
proportion 
of students 
used with & 
time spent) 



In general terms, how do you feel the 
approach is working out for you? Any 
problems? If so, what kinds? 



3: 



Are you currently looking for ideas 
or information about this approach? 
If so, what kind? For what purpose? 



Originally, using this approach meant 
that : (description) . Is this how you 
use it? Have you made any chaxiges 
recently? viiat? V/hy? How recently? 
Are you planning any other changes? 



'I'/hat do you see as the effects of 
this appi^oach? How do you know? Have 
yon received any student feedback? 





Do you 'have 
any pl^axis 
to use this 
approach in 
the future? 
If so, when? 



RECYCLE 
OR END 



How long did you use it? How 
did you implement it in your 
classroom? \^at problems did 
you find? What effects did 
it seem to have on students? 



Why did you stop using j.t? 



V/hen you assess the approach 
at this point in time, what 
do you see as its streng'ths 
and v/eaknesses? 



RECYCLE 

OR. END 



Do you 'work with 
the other teachers 
who are using this 
approach? How? How 
often? Have you 
made- any changes 
based on your work 
together? What? 




RECYCIiC 
OR END 



Rating the. Interview ' ^ * 

The Level of- Implementation rating form (Fif^e 5) designed 
to assist the rater in processing the information gathered from the 
interview. The i'ntervieV may be considered to consist of several seg- 
ments. The first segment relates to the use of the overall curriculum 
package. Subsequent segments refer to the use of the various innovative 
features of interest. The aim of the tating procedure is to assign to 
the interviewee separate levels and degrees of use for each segment. 

Letexinining the Loir for each seg;aaent is relatively strai£^t for- 
ward. The rater assigns a particular LoT-T based on the definition of the 
various LoU categories and the interviewee* s state^ments during th2,t se^ 
ment of the interview, _ ^ | 

If the interviewee is using the program or innovative feature, I 
the rater must also determine the LoU-Mod rating- for tlie interview seg- 
ment. This is somewhat more difficult since thWe are three behavioral 
categories to consider: information seeking, evalixative behavior, and 
status reporting. A separate LoU-Hod rating is made for each of these 
behavioral categories and then an overall LoU-Mod. rating is made for the 
inter\"iew segment. The suggested procedure is as follov;s: v/hen a state- 
ment is ma^e that appears to place the interviewee at a certain level of 
use for a parbiculai* behcivioral category, a tally mark is made on -the 
rating sheet by the appropriate niuaber. If the statement m.ay be inter- 
preted as applying to more than one level of use, separate tally marks 
are made for each. Once ^ the inte^.view segment is over, the rater must 
assign a LoU-Mod rating for each behavioral category and decide on an 
overall rating. These decisions should not be based solely on v;liich 
LoU~I4od has the raosb tally marks since one statement may be more signif- 
icant than the others. However, the tally marks do help provide evidenci 
for a d'ecision. In making these decisions, the rater must remember that 
each behavioral categx^ry and the overall LoU-Mod should be rated indepen- 
dently, Of-feen an individual will have different LoU-Mod' s ■ for the dif- 
fere^nt behavioral cate^ries. 

Two special designations are -provide!* at the bottom of the rating 
sheet, KD and NI. The ND rating is used for those sitiuitions in which 
the interviewee reports doing nothing in that behavioral category, Tlie 
NI rating is used to indicate when there is not su^fficient information 
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Figure 5: Level of Implementation Rating Sheet \ 



Interview Data: 



s\t 



Date: / / 



Tape .//_ 
ID // 



Do I Matrix fef: 



Interv Lewer : 
Rating Data : 

Date: ./ •/ Rater if I: 

Date: / / Rater //2 : 



How dirficult was this tape to r^lte? 
12 3 4 5 



Very 
easy 



Very 
hard 



DoU~0 
Non^llse 




LoU-Mod 


uo u— i 
Outside 




A 

MBCH 


B 

ROUT 


C 

REFD 


D 

MODF 


DoU-2 
Partial 












DoU-3 
Regular 












DoU-4 
Integr . 













Do I rating of : 



(program or feature name) 



DoU (Circle one)] 


LoU-MocI 


Info. 

Seeking 


Eval . 
Behavior 


Status 
Report 


Overall 


0 - Nun-Use ^ 


Mechanical. 


A 


A 


A 


A.. 


1 - Outside 


Routine 


B 


B 


B 


B 


2 - Partial 


Refined 


C 


C 


C. 


C 


3 - Regular ^ 

4 - In t e gra t Ive 


Modi f led 


D 


D 


D 


D 


Not Doing 


ND 


ND 


ND 




Past User? Y N 


Ko Info. 


NT 


NT 


NT 





Dol rating of: 



(feature name) 



. DoU (Circle one) 


LoU-Mod 


Info. 

S,e eking 


Eval . 
Behavior 


Status 
Report 


Overall 


0 - Non-Use 


Mechanical 


A 


A 


A 


A 


1 - Outside 


Rout ine 


B 


B 


B 


B 


2 - Partial 


Re fined 


C 


C 


C 


C 


3 - Regular 


Modified 


D 


D 


D ^ 


D 


4 - Integrative 


Not Doing 


ND 


ND 


ND 




Past User? Y N 


No Info. 


NI 


NI 


NI 




Dol rating of: 




(feature name) 






DoU (Circle one) 


LoU-Mod 


InCo. 
Seeking 


Eval . 
Behavior 


Status 
Report 


Overall 


0 - Non-Use 


Mechanical 


A 


A 


A 


A 


1 - Outside 


Roil t ine 


B 


B 


B 


B 


2 - Partial 


Ref i ned 


C 


C 


C 


C 


3 - Regular 


Modi Cied 


D 


D 


U_ 


D 


4 - Integrative 


Not Doing 


ND 


ND 


ND 




^Past User? Y N 


No Info. 


NI 


NI 


.NI 
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provided in the interview to make a rating. Either the^ interviewer 
fa/Lled to ask. for tlrfe information^or the- interviewee e did not provide 
e^oxi^^ information. 

' / ' Once the DoU and the overall LoU-Mod have been detenriined for a 
]5^articiLLar interview segment, they can "be recorded in the matrix on the 
rating sheet. If code numbers or letters are used to desi^ate the vari- 
jous interview segments, all the DolT's and LoU-Nod^s for an interview can 
fbe represented on a single level of implementation matrix to provide an 
implementation profile for that individual. This. matrix is also usefiil 
in compi:ij^ing ^data from several individuals. The completed Lol rating 
form for the sample interview (Appendix l) is given in. Appendix 2. 

:ilization of Level of Im-plementation Data 

ResQarch^^rs studying the processes of implementation will be 
primarily inte^/ested in individual users' level of implementation (LoI) 
profiles and how these change as a function of time. Hall et al. (l975) 
have\^ hypothesized that gtrowth in the q-iifality of use of an innovation is 
developmental for most individuals and the level of use should increase 
as a Amotion of time, 

^{Tliose researchers who engaged in sxommative evaluations or compara- 
tive st"ddics of curricula may simply be interested in assessing the "pur- 
ity" of "Dhe experimental and control groups. In these cases, the indiv- 
idual Lol Y^-triges are of little value and further data reduction is 
possible. vUhis may be done by producing master Lol matrices. One for th 
overall Lol \)f the program and one for each of the imiovative features 
assessed. E^h cell of these matrices v;ould contain frequency counts of 
the numbers o:^^ users in either the control or the experimental gxoup 
ha\ang a particular combination of DoU and LoU-Mod. Id(^ally all of the 
individuals in i:iie control group would be non-users wiiile all those in 
the experimental ^oup\would have a routine or refined level of use and 
be regular users. \^Researchers who find considerable variation in the 
level and degi^ee of\use of the various innovative features may be able 
to devise a research\deaign that allows the assessment of the relative 
contribution of\the various features on the overall impact of the curric- 
.ulxirn package. Tli^s tyj^e of research is rarely done and definitive result 
would prove invaluable instructional designers. 
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Application of the MethodoloCT" 

The preceding methodology has been used to measure the level of 
implementation of the junior hi^ ^chool^ science progo^am The Natural 
World (TNW) (Burkman et al., 1975-7). This is the revision and modular 
adaptation of the JSCS program; Probing the Natural World (intermediate 
Science Curriculma Study, 1970-72). The degree of implementation of 
the overall program and three separate innovative features were assessed. 
The various separate innovations are defined in Fignxre 6. These defini- 

Figure 6: Definiti,ons of Features of the TNW Program' 



Features ' Definitions 



Self Pacing (p) 


The rate at which each student or small group (4 o=c 
less) progresses is self determined. The teacher 
may encqurage slow or fast students to work at a 
more -reasonable rate. The use of a time schedule 
that students ^must follow is a substantial modifica- 
tion. Lock stepping the entire class constitutes 
non-use. 


Self Sequencing (s) 


Individual students select which module to start on 
and the sequence of modules they use. Teacher re- 
strictions on the number of students working on a 
module simultaneously are penaitted as are teacher 
siiggestions as to which module meets a student's 
needs or interests. Forming permanent lab groups 
that must jointly plan a sequence constitutes sub- 
stantial modification. .Prespecifying sequences of 
modules for more than 1 or 2 students constitutes 
non-ude. 'sr-^ 


Self Evaluation 


(E) 


Prior to taking the standardized test for a module 
students use the self evaluation items to assess 
their progress. ^ Students should correct their own 
items and decide whicja" activities, if any, to redo. 
Teacher scoring of the items is permitted. Using 
the results for grading purposes or requiring a 
student to redo activities based on the results con- 
stitutes substantial modification. 



tions were arrived at based on the author's personal experience with the 
program in both the development and implementation phase and from a 
study -of various Individualized Teacher Preparation modules developed 
for the original ISCS progx'am (intermediate Science Curriculum Study, 
1972-74)- 
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Subjects p , - . 

A total pT five- teachers using the TlVd progxajli v/are^ intervie\^^d, 
Three were teaching in a local school and t\fb vere from different loca- 
tions in the United States.'- All of the subjects were experienced 
teachers hkvirig taught science for nore^han five years. For two of the 
teachers this v/as- their first year of using the progTaiQ. 



Proceduxe 

The teachers v;ere interviewed- la^te in the school year to ensure 
that their use of the program had stabilized. The interview format fol- 
lowed the one previously described. Tlrie local teachers were interviewed 
personally. The U.S. teachers v/ere interviewed by telephone. Loucks, 
Newlove, and Hall (1975, pp 50-59) indicate that telephone interviewing 
is an acceptable procedure and may reduce the inhibitions present in a 
face-to-face interviewing situation.. In each case the interview was 
tape recorded. An annotated transcript ■' of one of the interviews is pro- 
vided in Appendix 1. The Lol ratings v;ere done by the author. 

\ . 

FindJLngs 

The Lol ratings of tlie five teachers for the overall program and 
the tliree innovative features are presented in Pigiu^e The individual 
teacher* 3 r^itings are identified using lower case letters. An examina- 
tion of the matrix for the overall program indicates that''''all ' the teacfte^^ ' " 

Figux^e 7: Observed Dol Natricies for the TlJW Prograin 



Dol Matrix for Overall Program 



Dol Matrix for Self ^^acing 



DoU-O 
Non-Use 




LoU- 


-Wod 






DoU-O 
Non-Use 




LoU- 


-I-Iod 




DoU-1 
Outside 




A 

I4ECII 


B 

ROUT 


G 

REFD 


D 

MODF 




DoU-1 

Outside 




A 

men 


3 

ROUT 


C 

REFD 


D 

MODF 


DoU-2 

Partial 














DoU-2 ^ 
Partial. 












DoU-5 
Regular 






acd 


be 


A- 




DoU-5 

Regular 






ad 


b 


e 


DoU-4 
IntegT. 


) 












^DoU-4 
' Integr. 


i>?-^ 

/ 




c 







!Pigup?e 7 (continued) 



Dol Matrix for Self Sequencing 



Dol JIatrix for Self Evaluation 



DoU-0 
•Non-Use^ 


' e 


' LoU-Mod 


DoU-1 
Outside 




A 

MEan 


B 

ROUT 


C 

REIT) 


D 

mm 


DoU-2 
Partial 












DoU-5 
Regular 






acd 


t 




DoU-<4 
Integr. 













Non-^Use 




— — ■ 1 — • — - — . 

LoU.'^Hod 




DoU-1 , • 
Outside 




A 

MECH 


ROUT 


c 

KEET) 


D 

MODE 


DoU-2' 
Partial 










■ — ^ 


DoU-3 
Regular 






ad 






Doir-4 
Integr. 













are regular users of the program and function at a routine or refined level 
However, the other matrices indicate that there are considerable varia^ 
tions in the teachers degree and level of use of the different innovative 
features of the program. 

It is always^ risky to generalize on the basis of the very limited 
data base. However, it would appear that there are significant differ- 
ences in hov; teachers implement a curriculimi package in the classroom 
and that an assessment of their overall level of use of the program is , 
inadequate to detect these differences. Further research is needed to 
substantiate the existence of these variations, to refine the techniques 
of measuring them, to determine whether they have impo3rtant effects on 
student outcomes, and develop methods' of helping teachers achieve those 
levels and degrees of use that maximize student outcomes. 
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APrENDTX I 



T :i p e s c r i \> t o f L t.> I T n t e r v i w ( 



The tapescrl.pt thai: follows 'iLs '<\ 1. iMUfir. r i pt of the actual intorvio\j 
of teaclu-r *'b". Tl;e interviev> format follovrs that of FLp,uro A (juite 
closely but not exact'! y. Som*^^ quest ions vrerc clropj>ed during ^^'he 
course of the interview because the recfXrired informaticui had b<>en 
give'u in an earlier part of the iuterv^ev^^. CoiuinenLs are lU'ovided to 
assi^^t tile reader in eate^^or i ng the Lnfonnat ion ]>rov:idc'd by llu^ 
various responses. Where a comment imlicat^vs a specific [,oU~Mod or 
DoU cat(\i.>ory, taldy.^mark was; recorded on the \.o\ \\:\\. 'iw\\ Shed . The 
completed hoi Rating Sheet ic. given in Ap]>cnelix 2. 



2. 



22 



I.trtervxcw with Teac her 

Int orv lew 

I: Are you using the modular voi\s ion of 



Tho Natu ral World program? 



R: 
1: 

R: 

I : 



Yes, with all 



■Do you use it rc^;ularly with 
studentsf \ 



^;ix claf^scs 1 teach. 

a 1 1 your 



Yes, everyone is on thi.s program. 

rally speaking how do you tlrink 
e pro^^ram is workinj; out in your 



( li 



s 1 tuat 1 on y ^ 

I riud it good. It works well. The 
kids are getting quite a bit out of 



it. They seem to 
Any probloms? 



like it 



U: Not with the program it.self, no. 
Possibly with class size. The 
program is good. 

I : Have you made any cdiang^es recently 
iu how you go nh(nit using the 
}>roga"*im? 

R; V/ith one" class, the weakest class, 
I sort of pacHHl them a bit at the 
beginning for tht* first book or two 
^tio t lu*y got pc*riiaps a bit moro of 

understanding of what their 
respous lb i hi t ios wore iu the program. 
But., aftor 'that they wert^ on their 
own . 

I: Wily did you mako this chang/-? 

K: Well/, t'lioy're a weaker class and 
reading, I think, is part of tluMr 
problem, I'lu^y didn't know what [o . 
look for. They wer^eu't reading it 
and understand Lug it the way tbey 
should before tbey startiul the 
oxpfc* r i ine»nt .s . 

I: Wh^it do you see as tlu* t»ffec^ts of 
the program? 

K: Wt^Ll, i think the studcnUs are a hit 
more capable now of doin^^, individual 
work - of <oHi»cting and organizing 
Information. Perhaps they're putting 
a ])it more faith iu their own alUtity 
to do ( h Ing.s . 



Comments 



To determine whethex a user or 
non-user of the program. 

Ind i eates extensive use . 

Probing for extent 'of use. 

Indicates regular use (DoU 3) . 

Probing status reporting LoU~Mod 
behavioral category . 

Reports program is going well 
with few indications of problems 
(LoU-Mod S.R. >- routine use). 

Probe status report ing. 

Again, indicates no problems. 
(LoU~^k)d S.R. routine use).. 

Probe' for changes that would be 
indicative of refined or modified 
use. 

The teacher has made some mod- 
ification of the program. This 
ma y i n d i c a t e me c h an ic a 1 u s e o r ' 
refined use. The change is not ^ 
large enough to be considered a 
substant ial mod i f ica t i on . f 
(LolW^od S. R. - refined) 

Prolu' to separate mechanical 
from refined use. 

(;hange was made to meet student, 
rathe r than t eaclu'r , needs and 
was designed to in(\reasi' the 
program's impact on students. 
Strong and clear Lndication of 
ref ined hoU-Mod . (I,oU~Mod EA\. - 
i'i»f inml use) 



Probe of evaluativi* 
J.olJ->k)d (^ntegory . 



behav i o rs 



lndtcatt^s an awareness of both 
the !^hort and long term changes 
in studiMU beliaviors, (holI~Mod 
I'MU refined u^;e) . 



Iiitervi ew 



f: How do you know this? 



Comment vS • 



]<: V.'eil, £rom the t:yp^ of ro.sa.ltr>, the 
type of quest ions, t-n^-^y ask, the way 
they sGt--up and organise, their 
e(|u Ipment 

■J: Have you gottcm any stuch^nt. foedhaek 
■ahiuit th<:^ program? 

R; Yc\s . J/ f in*] jj^ey like it. Ksperially 
l]\v hettLT students w^o recall y enjoy 
this type of t h i '\^ht' re tht^y^rt.- no I 
1,1 vd doun to the r. lower ones and thoy 
l\o.t a lot (Hit c^f 1 t . 

I: Aro you currently loo.lj;i^\^^^ for any new 
ideas or informat ion ahout the pro- 
ram? 

l^T^No, not really. .1 L is p,o.Lnj; wo hi now. 



J.; You've p^Lvcn me a pretty rc^asonal^l c^- 

ido.) ol how you iwv usin^; tlie pro-* 

Kiani in p,c^w(M*al tonus. Now let's 

look at somo of the' speeific\s. 

l.p 1 for t lu' Ovf^rall Pro^'a'nni: 

holf- i Kor,ular^ us, or of" tht: jM'o grain 

hoU-M<ul: 

1 . vS . - 1 n forma ti cm .^".L'L'kinp, - RoutiiVL^ 

H,H. - kvaiuat ivi' behavior - Kofiniul 

S,[^, - Status rt^[H)rtin^\ - Refinod 

Overall kolI-Mod rating', - Rc^fined 



Probe of evaluative- beliavior. 

Indicates informal evaluation 
of long term student outcomes, 
(LoU-Mod E.B. ~ refined) , 



Probe of evaluative behavior 
and has i s for chan[^es made . 

Generally aware of students* 
f o e 1 i n g s . 1 nd 1 c a t s some i n - 
formal evaluation. (Loll-Mod - 
K.B. - rc^ fined) . 

Probe information seeking 
category. ^ 

Indicates routiuL' l(/vel of use 
"for this category X (LoU-Mod I,S. 
routine ) . 

Transition to focus ing'^on the 
spcu^. iflc innovative features of 
the prog reaii . 



fat e fv i ew 



L: Our ft^^aturr of the proj^ram' that you 

havf mentiontul al rtMdy is self-pac- ■V^lie feature, 
iuj', . Th i ^; mt^ans tliat t lu*' studtuits^ 
eithoi- by themsolvL's or in small 
j^roiipsv. p.et to deride how fast they ^ .i^t''^ ^ 
gL) throu[',h the material. 
Art' you u.sin^', this, t yj>e of .st^lf- 
pac i nj.; w i t h your c 1 asses? ^ 

Yt\s , 1 am. 



C'onijneji^s 

1-fft roLfuc t i on and definition of 



Sepa'oiit^B&f: nsers and non-users. 



1: Do y(ni u^u^ It regularly with all the 
sl udtait Ji? \ (pau.se) You ment ioned tlu^ 
otu^ |\roup\ . , • " 



J nd I eates ur.i^ of the feature- 
Probe of degree c^f use. 

Indi(*ales Tegular use (DolI-3) . 



R: Ye.s, t lu* ouv j'^i'mij). I kept t lua^ , ^ . 

lc)g,ether for the fist t we ho(^kj;, Mina*.,Thi,s rc\sponse and previcnis onOvS 



9'' 

1 



24 



Interview 



Comments 



to get them to read things, to look 
at thelr|pn and to see 

really wrnrt the information means. 
, But, after that, after the first two 
books, I let them go tind I found that 
tliey were doing probably a>s well as 
some of the other jcla^sea^. - 

1^ In general terms, how do you see 
self pacing as working out? 

R:^ I think it works out quite well, if 
your students are capable of reading 
and know what they want to do. The 
.bulk of the classes have no problems 
at all. It works well. 



Are you looking for any new ideas or 
information about self pacing? 



R: 
I: 

R: 



I: 



R: 



indicate an overall^ref ined 
LoU-Mod for self 7pacing. The 
change is not major and was* 
implementgj^ to meet student 
and not teacher needs. The 
status reporting category is 
rated as refined. (LoU-Mod S.R. 
- refined) . 

Further probe OtI status report- 
ing. 

Indicates no problems which 
would normally indicate a 
routine level of use. However, 
states qualifications that 
imply assessment of program and 
students and the need for match- 
ing. (LoU-Mod E.B. - refined & 
LoU-Mod S.R. - refined) 

Probe of information seeking 
category. 



(pause) I don't follow you. 

Are -you looking fof new ideas or are 
you satisfied with how things are 
going now? 

Yea, as I said I've been pretty 
satisfied with it. 

Originally, the self paced approach 
meant that each student decided for 
himself his own rate of progress. Is 
this how you use it? 

Yes, they work on their own. I will 
from time to time put little notes 
on the board a^ to where I feel they 
should be at a particular time of 
year. After a month I might say you 
should be on your second book. And, 
they may or may not be at that point;. 
It gives them some way of pacing 
their time. 

Are you planning any more changes in 
how you use it? 

Not really. I might in another year 
spend a little more time on the 
Resource Book as I did this last year. 
I found that going through the Resource 
Book itself gave them a bit of infor- 
raation about chemical symbols. 



Clarification. 



Indicates no seeking of new ideas 
(LoU-Mod I.S. - routine) 

Probe' for any other changes. 



Indicates only minor changes but 
changes that are student , rather 
than teacher, oriented. (LoU-Mod 
S.R. - refined) 



Probe for planning for future 
changes^^ 

No indicVtiof/of specific major 
changes. The change indicated 
again has a student focus. 



Interview 



Comments 



chemical reactions and* things like 
that before they start into the 
program helps. 



I: What sort of effeclis do you see self 
pacing as having? 

R: Well, as I mentioned, the students 
are more assured of themselves. They 
have a little more confidence in 
what they are doing. They do some- 
thing on their o\m and see it is 
right, it's good - they've got a 
positive result. I think thi6 builds 
up a little more confidence in the 
student. They are not simply copy- 
ing notes or something like that. 
They are achieving some goal. 

I: Have you gotten any student feedback 
about self pacing? 

R: I found my s tuden ts really 1 ike th is . 
In one class, for example, I found 
about one-quarter of them have 
finished the program and are started 
into the grade nine program. They 
liked it. They worked well and they 
saw this as a goal: that they could 
finish and go on. They weren't locked 
in now for a month with nothing to do » 
, They could move into the third area of 
- the program. 

T: Do the students you teach encounter 
this self pacing approach in any of 
the other classes they take? 

R: Not really, no. 



Note: This is the onLy point in 
the interview where one particular 
modification is mentioned - this 
teacher has all the students work 
through the Resource Book before 
starting the program proper. This 
is a substantial modification of 
the use of the Resource Book but, 
since the use of this feature was 
not being-assessed,^ this was not 
followed up. However, th;i.sfdoes 
indicate the importance of doing 
the features separately and in 
detail. / " -^l^ 

Probe o f evalua five behaviors . 

Indicates some informal ^assessment 
of student outcomes . (LoU-Mod E.B, 
- ref,incd) 



Further probe of evaluating 
beh^ivior . 

Indicates awareness and some 
informal evaluation of students' 
affective states. (LoU-Mod E.B. 
- refined) . 



Separate integrative users 



Not an integrative user 



lol for the Self Pa'cing Feature : 
DoU-3 Regular user of self pacing 
1-oU-Mod: Overall rating - Refined 



i 

/ ■ ' ^ ^ 

I ntervlev 

I: Another feature of the program is 
self sequencing. Tliis moans that 
the students can decide which modulo 
to start on and what sequence to go 
through them. 
Do you use this approach? 

R: Yes. I use it on six of the seven 
modules. One of the module.s, I 
jEound , took a lot of care in hand- 
ling chemicals, and accuracy is 
important. So I prefer that they 
had done thre6 or four of the other 
modules before I allowed them to go 
on to that one. But,. with the other 
six it was entirely up to them what 
sequence they went in. 

I:. So, with that one exception, you were 
using it with everyone in the class. 
You didn't have some' students that 
. had a particular sequence that they 
had to go through? 

R: No. Except for the one class that I 
mentioned that started on one module, 
they all selected the sequence ^ 



I: \'Jhat did you see as the effect of 
self selecting the sequence? (|iause) 
Or, didn't you see that it had much 
effect? 

R: T really don't think it had too much 
effect. -Like the students looked at 
taking the easy ones first and that 
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Comments 

Introduction and definition of 
self sequencing. 



Probe to separate users and noriL- 

users. 

Indicates probably a regular 
user. Modification based on 
assessment of long terra needs 
and ou t^comes and is design eti 
to increase the impact of the - 
program on the students. ' " ' ' 
(DoU-3) (LoU-Mod S.R. - refined; 
E.B. - refined) . 



Probe for degree of use and for 
changes . 



Routine use (DoU-3) definitely 
indicated . Reference to change 
mentioned earlier. That change 
was student oriented to increase 
impact.. (LoU-Mod S.^. - refined). 

Probe 'for status repor.t. 

Indicates feature operating well 
which is characteristic of the' 
routine level of uee. (LoU-Mod 
S.R. - routine) . \ 

Probe for information iseeking. 



Indicates no search for additional 
information. (LoU-Mod I.S. ~ 
routine). 

Probe of evaluative behavior. 



Indicates that no evaluation of 
this feature had taken place or 
the evaluation indicated that it 



I: In general terms, how do you think 
this self sequencing worked out? 

R: Well, I didn't see any problem with 
it. There was nothing where one book 
had a great dependance on another 
one. So there was no problem with 
sequencing. 

I: Have you been hunting for any ideas 
or information about how to change 
or modify what you're doing? 

R: No, I really didn't see any need to 
modify it. It was working quite well. 
I was pleased with it and the kiSs ^ 
were quite happy. 
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Interview 



Comments 



•sort of thing. But, they can't judge 
from the color of the cover or the 
name of the book whether it is easy 
or hard. And, I think they sort of 
gave up after 'the first one. They 
took what was available and what 
they wanted ta do. 

Do the students you teach run into 
this approach in their other classes? 

No they won't. Now they may to a 
small, extent in mathematics but I 
think the testing and so forth there 
has been pretty well laid out and 
they are lock-stepped most of the 
time. 



had little effect. M^i^y indicate 
either routine or refined level 
of use (LoU-Mod E. B . routine & 
refined) . 



/ 



Initial probe for integrative 



use. 



Indicates little Or no use in 
other classes so no opportunity 
for an integrative approach. 



Lol for the Self Sequencing Feature : 

QoU-3 Regular user of self sequencing (minor modifications only) 
LoU-Mod: Overall rating - refined 



Int^erview 



I: The last part of the program we 
want to talk about is the self 
evaluation part of the program. 
As you know, there are self 
evalua^tion questions at the end of 
each chapter. The original intent 
was for th^ student to evaluate 
- hi§ o\'7n progress and make decisions 
about whether he was ready to go on 
or he had to go back and ,redo some 
things. 

Are you making use of the Self ' 
evaluation part of the program? 

R: Yes , the students complete them 
after each unit. Then I ask them 
to check them and then brirtg them 
up to me. I'll look them over, talk 
about the questions with the student- 
which ones he got wrong and why - 
what it should have l^een and I try to 
get hira to reason why the answer he 
has is or isn't what*^ he had there. 
And, I think they accept this. They 
use this in studying for tests as 

^ well. It's to their advantage. They 
see this as well . 

I: Do you use this approach with all the 
students? 



Comments 



Introduction and definition of 
self evaluation. 



Probe to separate users from non- 
users . 

Indicates probable regular use but 
suggests that there may be a major 
change relat ing to strong teacher ' 
involvement in the decision making 
process about whether to proceed or. 
no t . 



Probe of degree of us^. 
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Interview 



CoTninents 



R: Yes. In the odd case I might have to 
ask the student ,^o go back and redo 
a particular experiment. But, in 
most '"^cases , the student sees he is 
all wrong and decides to go back 
and redo it and see where he made 
his mistakes. 

I'. In general, how do you think your 
self -evaluation approach works out. 

R: It works out quite well. Again, the 
students coming in they chapge their 
ideas as they go through, the' course.- 
If you evaluate them at different 
times you get entirely different 
results. I think by the end of the 
course they can give you a fairly 
good evaluation. ^ 

I: Are you looking for any ideas or 

information about the self eveluation 
aspect of the course. 

R: No, not really. 

I: Originally, the self; evaluation 
approach involved allowing the 
students to make their o\^ decisions 
whether to go on or back. Other 
teaphers use the self evaluation 
questions for grading purposes or 
check them over and decide for the 
st'udent whether or not the student 
should go on. Are you using the 
original approach or have you made 
some clianges? 

^R: I don't grade them on the self 

evaluations for- a term mark. I'll 
grade them on the book as a whole. 
I let them mark them and if there 
are errors I discuss them with the 
student and between us we will decide 
whether he should go back and do an 
experiment - whether he understands 
what went wrong. If he understands 
what was wrong and knows, sometimes 
. that is just as good as redoing the 
experiment. 

I: You have moved away from the students 
making their own decisions and now 
use joint decision making? 

R: Yes. 



Further indication of major 
modification of the original 
approach . If the teacher can 
not convince the student to 
redo a section, the student 
may be told to redo it. 
(DoU-3 regular user) 

Probe for status reporting and 
evaluation. 

Response seems to iimply that 
there are no pirobleins (LoU-Mod 
S.R. - routine) ,and some sort 
of evaluation that indicates 
student' progress in self assess- 
ment. (LoU-Mod E.B. - refined). 



Probe of information seeking. 



(LoU-Mocf I.S. - routine) 

Definition of the original 
approach and probe for any 
changes. The statement of 
possible variations was included 
to avoid giving the impression 
that the original approach was 
the only acceptable one. 



Indicates two major changes: 
Although the self evaluations 
are not given a separate term 
mark, they are part of the book 
that is considered and thus are 
used for grading purposes. Second 
the decision making is joint and 
not just under student control . 
(LoU-Mod - Modification::) 



Direct p'xblUcv.d^f ,-dccision making 
changes., 

(LoU-Mod - Modified) Definitely 
established serious modification, 
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Interview 



Comments 



I: What do .you see as the effects of 
. self evaluation? 

R: It seems to make the student more 

aware of what he is capable of doing, 
whaf he has produced. He knows 
whether he is producing what he is 
capable of. They know that copying 
the answers from the back of the book 
isn't doing them any good. And, it 
brings out perhaps the honesty in 
them. 

I: Have 'you gotten any student feed- 
back about the self evaluations? 

R: Not too much. I've had students who 
have questioned the answers in the- 
bopk quite a bit and they've come 
up with other possibilities. 

I: Is yours the only class in which 
your students encounter self 
evaluation? 

R: Well, apatt f rom^i the stience progranf, 
I think they use it a little bit in 
the phys ed department. 

I: Do you ^ver work together with the 
phys ed department about self 
evaluations? 

R: No. We haven't worked together that 
much. 



Probe of ^aluative behavior. 

Indicates knowledge of the long 
term o'^utcoraes of the program. 
(LoU-Mod E.B. - refined). 



purther probe of evaluative 
behavior. 



To detect possible integrative 
users . 



Probe of" possible;, integrative use. 



No integrative use. 



Lol for the Self Evaluation Feature : 

DoU-3 Regular user of Self evaluation (with major modifications) 
LqU-Mod; Overall rating - Modified 



NOTES : 

1. This interview lasted about 20 minutes.' 

2. At tho. time the interview was^ conducted, the teacher was completing his 
second year of working wLth the program. He has worked with a total of 
eleven classes of students (about 28 per class) in levels 1 and 2 of the 
program (grades 7 & jS) . 



APPENDIX 2^ 



Completed Lol Rating Form 



The Lol Rating Sheet that follows indicates the data recorded and th( 
Lol ratings, made^ for the interview with teacher "b". A complete ^ tran- 
script of the interview is provided in Appendix 1. ' 
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1P t^<^X-'v^cn^ Data : 
Site; O/ 



Coinpletiod Lovol of Iinplcmc-ntation Ratinj^ Shoet 

Dol Matrix for: T fsf VJ (^) 
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Completed T.cvel of Iinpleinentatioh Rating Sheet. 
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